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INFORMATION PROCESSING APPARATUS AND METHOD 

BACKGROUND OF THE INVENTION 
Field of the Invention 

The present invention relates to an information 
processing apparatus and method and, more particularly, 
to an information process for protecting the 
intellectual property right of input data. 
Related Background Art 

In recent years, as a method of transmitting a 
single bitstream by combining multimedia data 
containing a plurality of objects such as motion image 
data, audio data, and the like, so-called MPEG-4 
(Moving Picture Experts Group Phase 4) is standardized. 
The receiver (player) of MPEG-4 plays back, e.g., 
motion image scenes in association with audio data. In 
such MPEG-4 player, various use limits must be imposed 
on some or all data to attain copyright protection and 
the like. 

The aforementioned MPEG-4 datastream has a 
function of independently transmitting/receiving many 
video scenes and video objects on a single stream 
unlike in a conventional multimedia stream. As for 
audio data, many objects can be similarly restored from 
a single stream. That is, the MPEG-4 data stream 
contains BIFS (Binary Format for Scenes) that modifies 
VRML (Virtual Reality Modeling Language) as information 



for synthesizing these scenes. Note that BIFS is 
information that describes MPEG-4 scenes using binary 
data. 

Since individual objects required for synthesizing 
scenes are transmitted after they individually undergo 
optimal coding, they are also individually decoded by 
the decoder, and scenes are synthesized and played back 
by synchronizing the time axes of individual data with 
that used in the player in accordance with the 
description in BIFS. 

In this manner, since the MPEG-4 player 
synthesizes a plurality of objects, use limits based on 
the alleged copyright must be imposed on these objects. 
For this purpose, a system for obtaining authentication 
information pertaining to the use of copyrighted 
information in units of objects is proposed. 

In general, in order to realize copyright 
protection, the transmitter must execute a process for 
encrypting data to be transmitted or embedding 
copyright information as a digital watermark in data to 
be transmitted. 

The receiver, i.e., the player acquires 
information for decrypting (decoding) the encrypted 
data, or information required for authenticating the 
digital watermark when the user has paid a given fee 
for the copyrighted information, and decodes and plays 
back the data that have undergone the aforementioned 



process, and contain desired motion image data and 
audio data. Upon decrypting the encrypted data or 
authentication using the digital watermark, copyright 
protection is assured by limiting the number of times 
of copying of data or inhibiting a given object from 
being edited with other objects. 

In this manner, the receiver decodes and plays 
back the received multimedia data after decryption or 
authentication using the digital watermark. 

However, when copyright information is embedded as 
a digital watermark in multimedia data to protect the 
copyright, that copyright information embedded in the 
multimedia data cannot be extracted until the 
multimedia data is decoded. 

Decryption can be done before the multimedia data 
is decoded. However, authentication of the digital 
watermark can be done only after the multimedia data is 
decoded. For this reason, the playback control of 
multimedia data protected by the copyright information 
must be able to cope with encryption or digital 
watermarking . 

SUMMARY OF THE INVENTION 

Under the circumstances, it is an object of the 
present invention to provide an information processing 
apparatus and method which can reliably control 
playback of information data to protect the 



intellectual property right involved even when the 
intellectual property information has undergone an 
encryption process or the intellectual property 
information is embedded in the information data as a 
digital watermark to protect the intellectual property 
right . 

In order to achieve the above object, an 
information processing method/apparatus according to 
one preferred embodiment of the present invention is an 
information processing method/apparatus for 
demultiplexing object streams from a datastream which 
includes a plurality of object streams each having 
predetermined information, and decoding, synthesizing, 
and output ting the object streams, wherein the object 
streams are authenticated, and whether or not the 
playback control of a predetermined object stream is 
done before or after decoding of the predetermined 
object stream is determined in accordance with an 
authentication method and the authentication result. 

Also, an information processing method/apparatus 
according to another preferred embodiment of the 
present invention is an information processing 
method/apparatus for demultiplexing object streams from 
a datastream which includes a plurality of object 
streams each having predetermined information, scene 
description information for synthesizing information 
contained in the plurality of object streams, and 



management information for managing a copyright of the 
information, playing back each information, and 
synthesizing and outputting the information on the 
basis of the scene description information, wherein at 
least one object stream is authenticated on the basis 
of the management information, and whether the playback 
control of the object stream is done before or after 
decoding of the object stream is determined in 
accordance with an authentication method and the 
authentication result . 

Furthermore, an information processing 
method/apparatus according to another preferred 
embodiment of the present invention is an information 
processing method/apparatus wherein encoded information 
data, and management data used to protect an 
intellectual property right of the information data are 
input, an authentication method for the information 
data is discriminated on the basis of the management 
data, and whether the playback control of the encoded 
information data is done before or after decoding of 
the information data is controlled in accordance with a 
discrimination result of the discrimination step. 

Other objects, features and advantages of the 
invention will become apparent from the following 
detailed description taken in conjunction with the 
accompanying drawings . 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a block diagram showing the arrangement 
of the overall information processing apparatus 
according to an embodiment of the present invention; 
5 Fig. 2 is a block diagram showing the arrangement 

of an IPMP control unit 20 according to the embodiment 
of the present invention; 

Fig. 3 is a flow chart for explaining the 
operation of the IPMP control unit 20 according to the 
10 embodiment of the present invention; 
X; Fig. 4 is a flow chart for explaining the 

&l operation of a motion image decoding circuit 17 

t)i according to the embodiment of the present invention; 

Fig. 5 is a timing chart for explaining the 
p 15 concept of operations upon permitting motion image 

f|| playback in the processing according to the embodiment 

J 

pi of the present invention; and 

Fig. 6 is a timing chart for conceptually 
explaining the operations upon inhibiting motion image 
20 playback in the processing according to the embodiment 
of the present invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

The preferred embodiment of the present invention 
25 will be described in detail hereinafter with reference 

to the accompanying drawings. 

Fig. 1 is a schematic block diagram showing the 
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arrangement: of an MPEG-4 player according to an 
embodiment of the present invention. 

Referring to Fig. 1, a transmission line 1 is 
represented by various networks, and is a network on 
5 which an MPEG-4 bitstream is distributed in this 

embodiment. Note that the transmission line of this 
embodiment includes, e.g., a storage medium itself such 
as a magnetic tape, hard disk, DVD-RAM, or the like in 
addition to a so-called communication line. 
10 In the apparatus of this embodiment, upon 

yp receiving an MPEG-4 bitstream distributed from the 

yj network or played back from the storage medium, the 

fy received MPEG-4 bitstream is input to a demultiplexer 

jjjL 2 . The demultiplexer 2 demultiplexes the received 

p 15 MPEG-4 bitstream into scene description data including 

'J\ graphic data, audio object data, motion image object 

data, object description data, and the like, which are 
respectively input to sync, layer processing circuits 3 
to 6 . 

20 Note that the audio object data has undergone 

high-efficiency (compression) coding such as known CELP 
(Code Excited Linear Prediction) coding, TWINVQ 
(Transform domain Weighted INterleave Vector 
Quantization) coding, or the like. The motion object 

25 data has undergone high-efficiency coding by, e.g., 
known MPEG-2, H.263, or the like. Also, the object 
description data contains, e.g., animation, or the 
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like, and is similarly encoded in a format suitable for 
each description data. 

Since the scene description data including graphic 
data, audio object data, motion image object data, 
5 object description data, and the like, which are 

synchronized by the sync, layer processing circuits 3 
to 6 in accordance with time information called a time 
stamp appended to the bit stream, have undergone coding 
such as high-efficiency ( compression) coding, and the 

10 like, as described above, they are decoded by decoding 
circuits 15 to 18 in units of objects. 

Since this embodiment assumes an apparatus capable 
of decoding even when an MPEG-4 bitstream contains 
objects of two different types for each of audio 

15 object, motion image object and object description 
data, two sync, layer processing circuits and two 
decoding circuits are prepared in correspondence with 
the audio objects, motion image objects, and object 
description data. 

20 The audio object, motion image object and object 

description data that have been decoded by the decoding 
circuits 16, 17, and 18 are supplied to a scene 
synthesization circuit 30, and a scene is played back 
and the graphic data is processed on the basis of the 

25 scene description information decoded by the decoding 
circuit 15. A final multimedia data sequence obtained 
in this manner is supplied to an output equipment 31 
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represented by a display, printer, or the like, and is 
visualized. 

The received datastream in this embodiment is 
appended with IPMP ( Intellectual Property Management 
and Protection) information described in, e.g., 
Japanese Patent Application No. 10-295937 proposed 
previously, assuming that control for stopping or 
restarting playback of individual audio or motion image 
objects that form each scene is required for, e.g., 



10 copyright protection. 



In the apparatus of this embodiment, the IPMP 
y information is extracted by the demultiplexer 2, and is 

4^ supplied to an IPMP control unit 20. The IPMP control 

>s unit 20 cuts off a bitstream or accesses the decoding 

yj 15 circuits 16 to 18 at access points 10, 11, and 12, or 

25, 26, and 27 in Fig. 1 to stop their operations, as 
needed. In this manner, even when a bitstream 
including contents that require copyright protection is 
input, it can be prevented from being played back 
20 normally. 

Fig. 2 is a block diagram showing an example of 
the arrangement of the IPMP control unit 20 shown in 
Fig. 1. 

Referring to Fig. 2, a CPU 40 for arithmetic 
25 operation/control controls the IPMP control unit 20. A 
ROM 50 stores a permanent or fixed control program 50a, 
fixed parameters 50b, and the like. A RAM 60 is used 
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as a temporary storage of the CPU 40, and includes an 
information area 60a for storing information such as 
command data, time data, and the like, and a program 
load area 60b for storing programs loaded from an 
external storage device 70. The external storage 
device 70 comprises an external storage medium such as 
a floppy disk, CD-ROM, or the like, and stores programs 
to be loaded onto the program load area 60b. 

An input interface 80 receives IPMP information 
from the demultiplexer 2 and time information from the 
sync, layer processing circuits 4 to 6. An output 
interface 90 outputs an access point control signal to 
the access points 10 to 12, and a decoding circuit 
control signal to the decoding circuits 16 to 18. 

In this embodiment, the IPMP control unit 20 is 
depicted as an independent processor element. 
Alternatively, some or all functions of the MPEG-4 
player may be controlled by the CPU 40 or may be 
implemented by software, as is known to those who are 
skilled in the art. 

The operation of the MPEG-4 player according to 
this embodiment, especially, operation using IPMP 
information, will be described in detail below with 
reference to the flow charts shown in Figs. 3 and 4. 

Fig. 3 is a flow chart for explaining the 
operation of the IPMP control unit 20 when the MPEG-4 
player receives an MPEG-4 bitstream. Assume that the 



MPEG-4 bitstream is composed of a bitstream of motion 
image data in units of packets, and a bitstream of IPMP 
information pertaining to the motion image data, for 
the sake of simplicity. 

In step 200, the next packet is obtained form the 
bitstream of IPMP information demultiplexed by the 
demultiplexer 2. 

Whether or not the next packet is present is 
determined in step 201 to check if the stream has 
reached its end. If YES in step 201, the processing 
ends; otherwise, the flow advances to step 202. 

It is checked in accordance with IPMP information 
in the obtained packet in step 202 if that IPMP 
information and copyright information of corresponding 
motion image data are embedded as a digital watermark 
in the motion image data. 

The IPMP control unit 20 can make this decision by 
checking the format of the IPMP information. 

More specifically, the IPMP information includes a 
description of the following IPMP object type 
designation at the beginning of the information. When 
IPMPS_Type in this description is "0", copyright 
information is not embedded as a digital watermark; 
when IPMPS_Type is " 1" , copyright information is 
embedded as a digital watermark. This IPMPS_Type is 
set in advance by the copyright holder or contents 
sender, and the IPMP control unit determines by 



identifying IPMPS_Type in the IPMP information if the 
copyright information of corresponding motion image 
data is embedded as a digital watermark in that motion 
image data. 

If it is determined in step 202 that the copyright 
information is embedded as a digital watermark, the 
flow advances to step 203; otherwise, the flow advances 
to step 209 . 

In step 203, a startup notice including a 
requirement of notice after decoding is sent to the 
motion image decoding circuit 17 to start up the motion 
image decoding circuit (or to continue its operation), 
since the motion image data must be decoded and the 
copyright information embedded as a digital watermark 
must be extracted. Then, the flow advances to step 
204. In step 204, the control waits for a notice 
indicating completion of the decoding processing. Upon 
receiving that notice, the flow advances to step 205. 

In step 205, the copyright information embedded as 
a digital watermark is extracted from the motion image 
data decoded by the motion image decoding circuit 17, 
and authentication is done using the extracted 
copyright information and the IPMP information 
corresponding to that motion image data. 

It is checked in step 206 if authentication has 
succeeded. If YES in step 206, the control unit 
notifies the motion image decoding circuit 17 of 
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completion of authentication. On the other hand, if 
authentication has failed, since a scene must be 
inhibited from being played back, the flow advances to 
step 208 to control the access point 26 to stop data 
5 supply from the motion image decoding circuit 17 to the 
scene synthesization circuit 25. In this manner, when 
authentication has failed, motion image data can be 
inhibited from being played back. 

In step 212, the packet of the stream obtained in 
10 step 200 is released, and the flow returns to step 200 
yri to obtain the next packet. 

l 3 | If it is determined in step 202 that the copyright 

information is not embedded as a digital watermark 

. :? ; : 

(i.e., the copyright information is appended by a 
15 method other than digital watermarking or no copyright 
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information is appended), the flow advances to step 
209. 

If the copyright information is appended to the 
motion image data, authentication is done using the 

20 copyright information, and IPMP information 

corresponding to that motion image data. On the other 
hand, if no copyright information is appended, 
authentication is done using the IPMP information 
alone. It is then checked in step 210 if 

25 authentication has succeeded - 

If authentication has succeeded, since the motion 
image data need only be decoded, a startup notice which 
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does not include any requirement: of notice after 
decoding is sent to the motion image decoding circuit 
17 to start up the motion image decoding circuit 17 (or 
to continue its operation) in step 211 . The flow then 
5 advances to step 212. On the other hand, if 

authentication has failed, since motion image data need 
not be decoded, the flow jumps to step 212. 

Fig. 4 is a flow chart for explaining the 
operation of the motion image decoding circuit 17 when 
10 the MPEG-4 player receives an MPEG-4 bit stream. Assume 
that the format of the MPEG-4 bitstream is the same as 

f£ that in the case of Fig. 3 above. 

y i 

Upon receiving the startup notice from the IPMP 
control unit 20, the motion image decoding circuit 17 
¥ h 15 decodes a packet of a bitstream of motion image data 

O demultiplexed by the demultiplexer 2 in step 300 . 

jfjj Upon completion of decoding of the packet, the 

p flow advances to step 301 to check if the startup 

notice which is received from the IPMP control unit 20 
20 includes a requirement of notice after decoding. 

As has been explained in the description of 
Fig. 3, when the copyright information of motion image 
data is embedded as a digital watermark in the motion 
image data, since authentication must be done after 
25 decoding of the motion image data, the startup notice 
sent from the IPMP control unit 20 to the motion image 
decoding circuit 17 includes a requirement of notice 
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after decoding. On the other hand, when the copyright 
information of motion image data is not embedded as a 
digital watermark in the motion image data (i.e., the 
copyright information is appended by a method other 
5 than digital watermarking or no copyright information 
is appended ) , since authentication is complete before 
decoding of the motion image data, the startup notice 
sent from the IPMP control unit 20 to the motion image 
decoding circuit 17 does not include any requirement of 
10 notice after decoding, 
ifj:) In this case, if the startup notice received from 

y\ the IPMP control unit 20 includes a requirement of 

pi notice after decoding, the flow advance to step 302; 

= :i otherwise, the processing ends. 

It 3 " 

15 In step 302, the decoding circuit 17 notifies the 

M 

y IPMP control unit 20 of completion of motion image 

'^■f decoding, and the flow advances to step 303. Upon 

O receiving the notice, the IPMP control unit 20 executes 

authentication in step 205 described above. 
20 In step 303, the decoding circuit 17 waits for a 

notice of completion of authentication from the IPMP 
control unit 20. If it is determined in step 206 
described in Fig. 3 that authentication has succeeded, 
since a notice of completion of authentication is 
25 issued to the motion image decoding circuit 17 in step 
207, a scene synthesis process in the scene 
synthesization circuit is started in response to that 



notice of completion of authentication. However, if it 
is determined in step 206 that authentication has 
failed, since no notice of completion of authentication 
is sent to the motion image decoding circuit 17, the 
operation of the motion image decoding circuit 17 
halts. 

On the other hand, when means that implement the 
data processing method according to the present 
invention are provided in the form of a program, since 
it is a common practice to process streams of audio 
object data, motion image object data, IPMP 
information, and the like in independent threads, 
synchronization must be taken among these threads. In 
this embodiment, a mechanism such as known semaphores, 
event flags, and the like is used to synchronize the 
threads . 

For the purpose of better understanding of the 
operation of this embodiment, the operations of this 
embodiment will be conceptually explained below using 
the timing charts shown in Figs. 5 and 6. 

Fig. 5 is a timing chart when authentication using 
copyright information embedded as a digital watermark 
in motion image data has succeeded. 

In this case, an IPMP thread manages threads for 
processing data corresponding to IPMP information. 
Therefore, when IPMP information is appended to a 
motion image object, the operation of a motion image 
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decoding -thread ( to be referred to as a video thread 
hereinafter) is controlled by the IPMP thread. 

Upon receiving the IPMP information corresponding 
to motion image packet data in which copyright 
5 information is embedded as a digital watermark, the 

IPMP thread starts up the video thread, which performs 
decoding of the motion image data. 

Upon completion of decoding, the video thread 
sends a notice of decoding completion to the IPMP 
10 thread. 

jSJ The IPMP thread executes authentication using the 

yJ copyright information embedded as a digital watermark, 

H,l which can be extracted after decoding, and the IPMP 

yu information. When authentication has succeeded and 

S3 

p 15 playback of that motion image object is permitted, the 

Si IPMP thread sends a notice of authentication completion 

J;: to the video thread. Upon receiving the notice, the 

video thread restarts operation and supplies decoded 
motion image data to the scene synthesization circuit, 
20 thus allowing to play back the motion image data. 

Fig. 6 is a timing chart when authentication using 
copyright information embedded as a digital watermark 
in motion image data has failed. 

In the timing chart shown in Fig. 6, when the 
25 authentication in the IPMP thread has failed and 

playback of the motion image object is inhibited unlike 
in Fig. 5, no notice of authentication completion is 
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sent to the video thread, and the video thread stops 
its operation. The decoded motion image data is not 
supplied to the scene synthesization circuit, and 
motion image data is not played back. 

In this embodiment, the MPEG-4 bitstream is 
composed of a bitstream of motion image data in units 
of packets, and a bitstream of IPMP information 
pertaining to the motion image data. Also, IPMP 
information can be appended to respective object data 
that form the motion image data, and audio data in 
addition to the motion image data, and a target in 
which copyright information is embedded as a digital 
watermark is not limited to motion image data but may 
be respective object data that form the motion image 
data, and audio data. 

In this embodiment, when authentication has 
failed, playback of data is inhibited by two methods, 
i.e., a method of cutting off the access point and a 
method of stopping the operation of the decoding 
circuit. Of course, one or both of these methods may 
be used. 

Especially, when only the latter method is used, 
access points for cutting off streams need not be set. 
By stopping the operations of the decoding circuits 
themselves, the decoding circuits 15 to 18 can be 
prevented from supplying nonsensical data to the scene 
synthesization circuit 30, power and time savings in 



the decoding circuits 15 to 18 can be achieved, and 
operation errors of the scene synthesization circuit 
can be prevented more reliably. 

However, since this method cannot avoid data that 
have already been decoded by the decoding circuits 15 
to 18 from flowing out to the scene synthesization 
circuit, data playback can be inhibited more reliably 
by also cutting off streams at the access points. 

Note that the present invention may be applied to 
either a system constituted by a plurality of 
equipments (e.g., a host computer, an interface device, 
a reader, a printer, and the like), or an apparatus 
consisting of a single equipment (e.g., a copying 
machine, a facsimile apparatus, or the like). 

The objects of the present invention are also 
achieved by supplying a storage medium, which records a 
program code of a software program that can implement 
the functions of the above-mentioned embodiments to the 
system or apparatus, and reading out and executing the 
program code stored in the storage medium by a computer 
(or a CPU or MPU) of the system or apparatus. 

In this case, the program code itself read out 
from the storage medium implements the functions of the 
above-mentioned embodiments, and the storage medium 
which stores the program code constitutes the present 
invention. 

As the storage medium for supplying the program 
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code, for example, a floppy disk, hard disk, optical 
disk, magneto-optical disk, CD-ROM, CD-R, magnetic 
tape, nonvolatile memory card, ROM, and the like may be 
used. 

5 The functions of the above-mentioned embodiments 

may be implemented not only by executing the readout 
program code by the computer but also by some or all of 
actual processing operations executed by an OS 
(operating system) running on the computer on the basis 
10 of an instruction of the program code. 
*!;f Furthermore, the functions of the above-mentioned 

til embodiments may be implemented by some or all of actual 

f|j processing operations executed by a CPU or the like 

£T arranged in a function extension board or a function 

^ 15 extension unit, which is inserted in or connected to 

the computer, after the program code read out from the 

i u 

"J 1 storage medium is written in a memory of the extension 

O 

O board or unit. 

When the present invention is applied to the 
20 storage medium, the storage medium stores program codes 
corresponding to the above-mentioned flow charts. That 
is, program codes that include modules stored in the 
external storage device 70 shown in Fig. 2 are stored 
in the storage medium. 
25 As described above, according to the above 

embodiment, upon demultiplexing and playing back object 
streams from a datastream that contains a plurality of 



object streams each having predetermined information, 
playback control of object data protected by copyright 
information can be done not only before decoding of the 
data but also after decoding, and more flexible 
copyright authentication can be achieved. 

Especially, whether or not playback control of the 
object data is done before or after decoding of the 
data can be determined based on if copyright 
information is embedded in the object data as a digital 
watermark. 

In other words, the foregoing description of 
embodiments has been given for illustrative purposes 
only and not to be construed as imposing any limitation 
in every respect . 

The scope of the invention is, therefore, to be 
determined solely by the following claims and not 
limited by the text of the specifications and 
alterations made within a scope equivalent to the scope 
of the claims fall within the true spirit and scope of 
the invention. 



